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ABSTBACT 

\ \ . / The evaluation of an interdisciplinary, process 
curriculum for'real problem solving required a .tool for 
differentiating that p'rd'gram's application from the actual treatments 
, taken* by students receiving more traditional alternatives to the 
experimental program. Treatment? could not be controlled or assigned 
at random, and these restrictions un^ers'cored the need to. distinguish 
between programs in natural settings at the level of student learning 
activities. An observation instrument , ^'System to^On^derstand the 
Dynamics of Education in Non-Traditional- Settings tSTODENT)," was 
developed which enabled reliable, practical representation of the 
students' behaviors in both groups. Its ease of application and 
analysis of results were illustrated with data^ contrasting student 
behaviors in the experimental and conn?arison groups for^ two years of 
data collection. The results also reflected changes in-' application' of 
the experimental' program over the two years. (Author) 
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• Abstract 



TJhi^4i^valdation'^^ an interdisciplinary, process cur- 



.^^^lum-for real . problem solving required a tool for . 
'differentiating that program's appl^^^^ from the 
actual tifeatments taken by students receiving more , 
traditl^pn^I /alternatives to the experimental program. 
Treatments s could not be controlled or assigned at ran- 
dom, and these restrictions underscored the need to*>' J 
distinguish between programs in natural settings at 
the level of student learning activities. An observa- 

■ ■ ^ ■ ■ ■■■ • 7 ■ " ■ ■ -'^ • • 

tioh instrument was developed which enabled reliable,, 
practical (^presentation of the students/^ beh^i or s in 
both groups. Its ease of application and analysis of 
results were illustrated with data contra§ting student 
behavicyrs in the experimental and comparison groups 



for two years of data collection. "The" results also^ 



Teflectedchanges in application of the experimental 
program over two years. , - - ^ 
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ANALYSIS OF STUDEtir BEHAVIORS IN TRADITIONAL 



'7 - ^ . ...AND-.NONrl^pITJONAL PROGRAMS^„_^ 

■■■■ %■']■''•■' 

i V DCjfing the last' decade, the\ observation of subjects in natural 
settings has received cpnsiderable a^te^^ technique for 

data collection in educational researdb. In a^ single reference, Simon 
/and Boyer have compiled \:he most comiDrehensive. collection of observation 
/ insttuments in 'their '17^volume anthojggy^ for Behavior (1967, ' 

'1970a; 1970b) • with documentation »on 9? \qbservation systems, 73 of which 

■.'■■■■«.■■■ ' / • ' . \ i - , . . . 

" ■ . . / " "'■ \ '* * • • • • * 

were designed for classroorrv^ observation. 1 of the Educa-' 

tibnal Resources Informajdon Center (ERlu^i. files suggests that t,he number : 

of ^xi^ting instruments for systematic dbserVat ion of classroom instruc- .' 

- * ■ ■ ■ \'. r ■ , - 

'Hi. 

V } 



tion may well be/placed la the hundreds, \ . 

^yerpped , 



Confronting the chaos' which has deyerpped, Bosenshine and Furst 



(.1^73) offered' a very seasoned perspective op tj^e maze of instrumentation 
and research in their incisive critique of^ the state of the art..' With so 
many existifig systems which are underdeveippea, g^^^ and under- 

researched, Herbert and Atttidge (1975) -even uVgdd a halt to the creation 



H5lphew instruments without. :good„reasmsJ:oJusJt^^^ . 

l ■ * y 

Their guide for developers and users . of observation .systems stands out in 
the literature for its attention to the issue of practicality in the im- 
plementation of a system and the dissemination of iti results. Yet, a 
system- which is difficult to apply is not likely to y|ield. valid, measures, 
and one which^requires extensive observer . training ox| costly recording 
equipment is not. likely to be used a-t all. eyen undei fayorable circuip- 



" An^lysii of . . 

■'■3 . 

fstances for adoption, a system whosQ^H|^^^curDbersom§ Id assemblej an- , 
alyze , -inter jDret / and disseminate i M^ ^mBStK^ \ ^'^ usetf regui^arly 'b^r-, 
yond the descriptive element in thev resi^2MHRp endorsed by P0$enshiney^ 
and: Furs t"'( 1973, p. 127) for improNTing .educaliiW practice. . > 
• /The "^purposfe of .this art^^ an observation sys- 

tem (a) whose functions apparently could [^^^ served by prior extant^ . 
. sysiteais; (bii'-which is 'relatively easy and i4Te'xpehsive to^apply; (c) whose^' 

ata can be analyzed and interpreted' ffeadiiy by educational practit^^ 
wihtbyt special training^ or ecjuipment; and (d)^' whose resolts^enable po- - 
; teotialJLyHnsightful comparisons for teacher training or prqgtam evalua- 




tion. 



■ The Instrupient ' * ■ . . 

. Context' f or Oeveldpment . : ' \' . [ > ^ . r . ' J 

The neW^servation instrument v/as developed as a means, for com- 
paring the actual J.nstructiohal treatments received ^by a national sample 
of experimental. classes using an experiential -interdisciplinary currifcu- 
lum'in science and mathematic§ . f or real problem solvifig, and. ^•control" 
classes who were putsuing their regular instruction in science and. mathe- 
matics. Entitles a "System To .Understand the Dynamics pf Education 'in ^ 
Non-Traditional Settings," the ngme of the instrument was fitted to the 
acronym STUDENTS ^to emphasize that the focus of the observation schedule 
is what the students do, not what the teacher does. . In this, regard, 
Charters and Jone^. offered an important caveat to educational evaluiators- 
who fail to certify that treatments applied in 'experimental and control 
situations differed at the level' of student. learning activities: , - 
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' * The manifest;|)ur teacher's role performance 

/ * ^is 'to produce leaning in students, but this cannot 

* ^'•^ ; . 'Inappen directly. .The best the teacher can jdo is to • 

induce statementiS to engage in activities de^ed in- 
\ strunenttl to Jfche covert ^ processes he 

• ' hopes to^^ffect. It is the stud^^^^ own activities 

' / ? ' , ■ \ and experiences that are most- immediately related to 
* V learning outcomes... and. it is of no small importance 

for program evaluators to measure the school's educa- ^ 
. V tional program as experienced and enacted by students ^ 

• ' .. " . V- (Charters and Johes^ 1972, pp. 6-7). 

Determining the essence, of these differences between the experimental 
treatment and the comparison programs was a special concern for the evalu- 
ation of Unified Science and Mathematics for Elementary Schools (USMES) 
because of the student-centered nature of the USKES curriculum. 

' The UShCS project was formed in response 'to the recommendations 
of the 19^7 Cambridge Conference on the Correlation of Science and Mathe- 
matics in the Schools . Funded by grants from the National Science Founda- 
tion and coordinated by a staff at the Education Development Center in 
Newton, Massachusetts, the USMES project has purported to. develop the com- 
petencies of elementary shcool students for real, complex problem solving. 
The goal of the; USMES pro ject has been the development, and trial implemen- 



tation of approximately 3D interdisciplinary units engaging the students 
in long-range investigations of real and practical problems taken from ^ 
their school or coiranunity envitonrrients. By responding to these problems, 
called "challenges," the students are supposed to develop their problem- 
, solving abilities and to do so- in a manner that gives them an experiential 
. understanding (learning-by-dping) of the problem-solving process, as well 
.^ds the^cquisition of basic skills and concepts, particularly in the areas 
'of science and mathematics. Acting only as^a coordinator and collabora- 
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tor, the teacher is supposed to adopt a non-traditional, more indirect V y 
style of teaching^ Another emphasis of the program is that progress. to- 
ward a solution to a problem should require the combined efforts off' 
group of ^students not just an individual stu(^ent working alone/ While 
some wotkvlnay be done individually, the USMES approach provides for a:di- 

visiori^ of -labor and an exchange of ideas — a total group effort which 

* . ■ ■ ■* ■ 

should enhance students' socialization and cooperative spirit* 

Unlike more structured curricula which; might prescribe relatively 
unifbrm student and/or teacher activities through texts, workbooks, teach- 
er guides, programmed instruction, etc., USMES is purported to-be "an im-' 
portant new style of education," (Education Development Center, May. 19^4, 

jpl 1). While a series of challenge units and tangible resource' materials, 
havei been develdped for USMES, this program, according to its developers, 
is more accurately portrayed as a philosophy of education. than as 'a col- 

I lection of rpaterials. Each USMES challenge should evolve from the child- 
ren's identification of, and action on?,' a problem which- is real aod impor- 

■ tant to them. And so, by design, the USMES approach could result in as 
many different tFea^tment groups as there, are classes usirig USMES. Fur- 
thermore, the evaluatiog team could assume that classes in the "control" 

. group were homogenepus only with respect to their non-use of JJSMES. 

Since treatments and dosages could not be assigned or mahipulated 
by the evaluator, it was not enough to apply a pre/post control group de- 
sign to assess differences in students' cognitive and affective develop-^ 
ment. Tentative explanations for any observed differences in student perr 
formance heed to be stilidied as well. 




J ■ " * "fit 

philosophy is an eclectic one; if encompasses features of the theoretical 
p^s^tions expressed by Dewey / Bruner, ^agne, aYd others*. When STtDDENTS* 
was developed, and;^even at this writing, ' the U5h€S developers had not ar- 
ticuiate^a theoretical position op educating students for problem solving 
(Shann-, 1976). However,* most consistently evident in the USMES develop- 
ers* written Statements about the USMES approach were references which 
called to mind John Dewey's "five logically distinct, steps" of the -problem 
solving process, his philosophy of experimentalism, and Bruner 's (1960) 
conceptions^' of discovery and enquiry. ^ These sources provided a starting 
point ^for the development of STUD^S categories. * The form underwent suc- 

^sisive revisions and pilot testing over a period of th^ years. in both 

• • ' . -J . 

USMES and non-USMES classes. , ^ - ^ . 

TQ the best of the-;evaluator's knowledge,* no ofcher ^observation 
instrument could satisfy the considerations served by STUDENTS^ it could 
be used easily and inexpensively by trained observers to monitor students* 
verbal, behaviors' and non-verbal activities in bolth traditional and non- 
•traditional student-centered Classes at the elementary level. Despite the 
intended use; of the ihstfijinent in prloblem soTv^ .and in" more ' 

traditional, science and mathematics programs at the elementary level, . 
STUDENTS hasl also- been used effectively to contrast studer>t learning acti- 
vities during English instruction in public alternative high^ schools ver- 
sus traditional comprehensive high schools in Massachusetts (McNeil, 
.1977). However, researchers who wish to investigate teacher behavior^ 
more directly and/or to study situations in which verbal behavior is ex- 
pected to predominate should find other observation instruments more sen- 

sitive to their purposes* ' \ 

• .. . . ■ J 
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• : Several kinds of data were collected in an effort tc/ jifferenti- 

; ; ■ -/ ■ ■■ . ' . ^ ' . • ■ .-. ^' 

--ate the nature and intensityof treatments applied in the experimental and 

•■•*•■-'.•. . ^ > ^ » . . 

control gfdups (see Shann, 1975)* -Widely varied patterns of USMES use 
were reported by U5MES teachers. Furthermore, the distribution of average 
^ tildes reportedly spent on non-USMES school subjects and activities for 
USMES classes when Chey did not pursue USMES were very similar to the d'is- 
tribution of average times, reported for control classes, and for JDoth 
groups', 'the measures of variance in thege times were very large. Similar- 
ly, both' USMES and control groups represented tremendously variable.pat- 
terns in the kinds of non-USMES curriculum materials and programs they ♦ 

used. Indeed, there- was as much variability within groups 'as there was 

• \ • ■ ■ ■ ^ ■ • 

between treatments, except for the faict that control classes, had not used 

USMES.. Only the STUDENTS observation instrument was sensitive, enough tp 

abstract clearly distinrguishable differences in the kinds of activities 

pursued by USMES students working on USMES challenges versus control stu- . 

dents pursuing their regular science/mathematics instruction. 

Identifying. Characteristics 

As- its-title -suggest^y-STUDEI^S^^^^ 

learning activities of students in a non-traditional edocational program. 
Yet to enable comparisons between treatment groups for the USMES evalua- 
tion, the '^instrufnent had to afford opportunitites to'tajly evidence of ^ 
teacher directiveness and to represent what students were doing in sup- 

V posedly morjfe traditional alternative programs. 

Both theoretical, and practical considerations .guijjed the selec- 
tion of categories of behaviors to include in the instrument. The^,USMES 
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The final version of the STUDENTS form includes 29 categories of 
h4havi,ors grouped in four areas:, (a) nonverbal, .physical activities 0/ the 
xhild; ft)) verbi3i interactions of' the child— with other children, with the 
teacher, and with the. class as a whole; (c) more passive look/listen. be- 

!.'■■- • . • ' - ' • ^ ' . • 

haviors of the child; and (d) other behaviors, ?uch as maintenance, wait- 
ing, and fooling around^ In* Table 1, the 29 behaviors are listed as they 
.appear on the | form used for data collection, and in Table 2, the behavior-. 

alj categories-! are described with examples provided. 

' ! ■' ■ • * * 

Insert Tabies 1 and 2 about here 
a . ; , ' 



Use of STUDEI 



Observers must be trained in. the proper use of this observation 



instrument. U 
interest , the 



pbn entering the ^classroom for the instructional period of 
observer conducts seven bounds of observation. During each 
round, the obsisrver is j:o look at each chiid as if taking a snapshot, then 
place a tally If or each child in the behavior category on* the form which 
best describes! -what th4t child was just doing. , Thus, the number of ^^^"^ 
lies on each rjound should equal the number of students *in the class. Each 
round of observations aould take anywhere from a few seconds, if all* the 
'children are engaged in the same activity, to a maximgm of five minu1;es. 
However, to attain more uniform time samplings across classes and occa- 

signs of observation, the time period between the start of each round is 

*•■'■■.■ ' ♦ . 

set at five minutes. ^ 

y Persons lof professional, congenial, nbn- threatening demeanor. 

shoullL be seleq^d as observers, and the same person should be assigned 
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for repeated observations in sample classrooms so as t6. minimise the dis- 
ruptive or threatening effects on students and .teacher which an unfamiliar 
person^ can ha^on a classroom. Prior to training, observers should study 
the category descriptions and examples, and 'the investigator/training di- 
rector should pilot test the form to determine if the categories of behav- 
iors are representative and exhaustive 'of the behaviors to be observed in 
thei study. The STUDENTS form might be modified by4:he deletion of cate- 
gories or examples of behaviors not likely to be observted^n the new situ- 
ations, or- by the inclusion of new examples of behaviors more likely to be 
witnessed.- Categdry'^lO, "Reads 'How-to' cards,"* which was appropriate for 
the USMES evaluation, might be deleted or changed to "Consults reference 
material" such as a library book, encyclopedia, dictionary, etc., to lo- 
bate information or find out how to do something, rather than ask the 
teacher. ' ^ 

Videotapes of claates' thought to be representative of those which 

will be observed in the study should be obtained for ob^^ver training. 

I 

As. a group, observers should practice using the form (see Table 1) to re- 
cord the. behaviors ' they .witness on the tapes; The training director 
should rerun segments*of the tapes so that observers can compare tallies 
for the same behavi^s and resolve any dis'crepanciees in their choices of 
categories. -Once the observers can designate with ease and precision the 
categories for the behaviors they witness, they should practice applying 

the five-minute founds of observations with actual classes similar to 

- . *■ f , • 

those which will be used in the study. Ideally, this practice should take 
place with all the required observers viewing a class through mirrored 
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glass,. If this situation is not available for training, 4:he observers 
\ should practice in real classes, as unobtrusively as possibley in groups- 
of two's so that they can check the^ agreement of their lialli€S, by tally, 
not just total tallies.' Particularly when the observation data will be. „ 
used to compare programs, the observers should be directed to arrange com-> 
parable time slots within the day and within the week for observing the . 
classes undergoing different treatmen^ts. / . . / 

Reliability and Validity ' « * . ' I' 

' Jhe provisions and suggestions for training in the use of . 
STUDENTS vere made in consideration of the reliability and validity of the 
' data. Perfect intra-^ and inter-scorer reliability has been obtained - 
during the training/sessions with the^ STUDENTS form for the 1974-75 USMES 
data collection. The time sampling achieved in the u^se of seven five- 
minute rounds of observation in each instructional'period should enhance 
the reliability of the measures as far as their -stability is concerned. 

The time sampling procedure also addresses the issue ot^validity^ 
of the data. The tallies should reflect the distribution in amounts of 
time students devoted to" each: activity. More importantly, the behavi^or 
categories, as included or modified in the ^form should represent theTcinds 
of behaviors typically found in the classes bb be studied. This Assurance 
Should be achieved through pilot testing of the form ais suggested -earlier , 
^..-^s^^n^through content validation of the form by kndwledg&able judges. Other 
validity. evidence that this* observation system revealed student beha\)ior 

patterns. which could explain student effects of the USMES curriculum was 

. •, . > . ,. , ^ . • y _ 

. . Offered in the USMeS evaluation reports (Shann, 1975; Shann, et al , 19*^5). 
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Analysis of the Data' » ^ 

0* ... * 

. • the number of times each class should be observed, the number of 

.classes' to be observeV, the number of comparison groups to be represented 

~ thesTe and other consid^rations are issues if or the design of a particu- 

■"■ • ■ ■ ' * ■. ' . ' ■ ■ " „ ' . • ■ 

lar. study and the purpose of that particular investigation, in general, 

however, the analysis /of data from STUDENTS requir€S. only the averaging of 

'•• ' : • ^ 

^ta^ie^s across ^^-^^ or group of classes and the expres- 

sion.bf. fehfese%verage ^s percentages of total frequencies. 

The percentage of tallies recorded in each behavior category offer, then, 

an indication of thi^ amount of time students devoted to that behavior ca- 

tegory./ Even the unaltered raw data from a single recording form may be 

instructive to, investigatofs who wish to examine any changing patterns in 

• ^ -y.: : . ^ ■ ■ ' ■ : 

student. behaviors within a 35-m^nute time frame for a single class, arid* 
^more sophist^fcirted analyses of the data from students can, be accomplished,^ 
top, with relative easev ' V 

■■ ■ . .. - ■ ■ ■ • . • . ^ 

• . ' . lllustraUon Of Results . 

■ Result's from the 197;3-74 and 1974-75 school years of ^data collec- 
tion for the U.SMES~evaiuatior) have been selected to illustrate the appli- 
cation of STUDENTS.. For each year of-'data collection,^ the sample classes 
'were phosen to include a cross section -of elementary -grade levels, geo- 
graphic areas, and spcio-economic levels/ For the .197^-74 /data- shown, all 
USMES classes were tSughJ^by teachers newly trained in USMES at national 
workshops, while their c'ontrols for 1973-74 were- classes from the same . 
school, at tile satne grade level, who?e teachers had not been trained to 
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use USMES. The 1974-75 data* were based on classes of students new to 
.USh€S whose teachers included a cross ^section of new and experienced UStCS 
teachers. Controls for the J974-75 sample were matched with the USMES 
classes for grade. leveL, socioeconomic level, geographic area, and general 
features of the school program, but this year the controls were obtained* 
from neighboring schools where no USMES was used. ' 

''Observers were directed to apply the STUDENTS observation. tech- 
riique during the Fall, Winter, and Spring of the 1973-74! school year, with 
the specific dates to be^^orked out by the individual observers and their - 
patticipatin^ teachers . Observers were told to arrange specific dates for 
the 1974-75 classroprp qbservatlon schedule which would sample USMES class 
tirne' at the beginning, middle, and ehd of the unit's, however long rthey . 
were expected. tp lest, so that olDservatipn 'times in each USMES class w^re 
dependent on the duration of the unit. Both years, the observers were di- 
rected to arrange comparable time slots within a day and within a week for 
observing each USMES class and its corresponding control. USMES classes 
were to be observed while USMES was"^ going on; control classes were to be 
observed during their mathematics or scie^cp class periods. 

The evaluators had expected that the seasonal ^.distributioo of 
Fair, Winter, and Spring observations during 1973-74 would correspond 
roughly to the beginning, middle and end of USMES units, but this was not< 
the c^ise. Many classes completed their units in less than a year's time, 
and the USMES project developers no longer urge that;the duration of. a u- 
nit be one. school year.. The variable lengths of time' which-- the sample 
classes deVoted to USMES account in part for thaf attrition observed in 
the "n's". . ^ 
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Insert Tables 3 and 4 about here 



The figures shown in the cells of Tables 3 and 4 represent estimates of 
the percentages of t^ spent on each category of'behavior by UShCS and 
control. groups, at polQ^^ear the beginning, middle, and end^of USl^S 
unit' activity by USMES classes. ' 

Interpretatioa , 
'it was hypothesized that in the^US^ES mode of learning, the 
teacher would have to- adopt the role coordinator pr collabbrator,' ra- 
jther than the^director's role more typiwlly ac^pted. by classroom teach- 
ers. Students* using USME5 were expected- to engage in active, hands-oa, 
"learning by doing" as they pursued problem s|p.ving activities, working 
cooperatively with their peers and. relying less on their teachers for, in- 
formation and direction. By comparisoriVVchildren in control classes were 
expected to exhibit more passive, structured^ teacher-directed and 
teacher ^dominated behaviors. The STUDENTS ^observation data In Tables 3 - 



and. 4 offer some support for these hypotheses, but traditional teacher- 
practices were still evident in USMES classek^ ' . 

-Teachers continued to dominate class activities. In both USMES 
and control . classes . for both years of data collection, students spent the 
largest percentages of class time listening to and/or looking at the 
teacher. However, different patterns of teacher domination in USMES clas 
ses emerged for 1973-74 versus 1974-75. In 1973-74, there were seasonal 



/ 
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differences in the amounts of teacher-dominated student activity in USMES 
classes, as control classes exhibited a sustained focus on the teacher 
(see Table 3, line 26)) Control s,tubents spent fully 31% of their ob- 
served class time in the^ Fall listening to or looking at the teacher, 
while USMES students spent a much smaller percentage of their time (1485) 
in this way. The differences dropped from^a ratio of 2:1 in the .^all (31% 
control versus 14% USMES), to a ratio of 3:2 in the Winter (30% versus 
21%), to almost a 1:1 ratio in the Spring, with no appreciable difference 
between USEMS and control classes (28% versus 26%). These results sug-| 
gested that in the beginning the 1973-74 USMES teachers dXd in fact addpt 
less dominating roles. However, in the final stages of the- units ^/the 
USMES teachers dominated glass time to a much greater extent than they did 
earlierMn the school year, perhaps because they were addressing original 
student reports of what was learned in the units (see Table 3, line 19). 

- ' . Results from- the 1974-75^ data collection showed sustained em- 
phasis of 16% to 20% of observed^class time on listening to and/or looking 
at the teacher—for both USMES and control classes, the' teacher interview 
data also obtained as part' of the USMES evaluation, suggested an explana- 
tion for. this curious finding. Many teachers reported; that their. in- - 
structors at Summer 1973 USMES workshops had urged theinvoto *ref rain almost 
totally from directing their students' work on USMES. This report^ may not 
have been factual, ahd indeed the developers. acknowledged* the. importance 
of some teacher direction with USMES. Nevertheless, those teachers' per- 
ceptions were real. Owing perhaps to the USMES developers' efforts to ad- 
dress understandings abou^; Appropriate amounts of teacher directiveness in 
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USI4ES or to the experienced teacher's adaptation oN^he^USMES program, the 
1974-75 sample US^€S teachers expressed more and -more confidence about the' 
importance of teacher direction at certain critical points in their USMES 

n i 

units for more^fficieht student solutfon of USMES challenges. ^ Increased 
teacher directiveness In the USMES sample for ^974-75 may explain another 
. year-to-year difference in the patterns of USMES student, behaviors: in ; 
the 1973-74 data, the expected shifts weile found in the student activities'^ 
emphasized over the course of their units, but in the 1974-75 data, little 
variation could be seen in the patterns of student activities from one ob- 
servation to another. ' ^ , 

The data from. Table 3 suggests that during the Fall, 1973 obser- 
vation period, when USMES students beginning their units focused on their 
teachers to a relatively small extent, the USMES students were engaged in 
the "hands-on" activities related to preparation for, and, engagement ih, 
the data collection process for problem solving. Approximately 31SK of the 
observed class time was directed toward constructing, assembling, and 
testing/experimenting (summing lines 3, 4, and 5). Calculating and re-, 
cording data consumed another 1485 of their time (lines 6 and 7). However, 
the percentages of. time spent on these data collection and odata management 
activities by USMES 'classes tended to diminish considerably from Fall to 
Winter and from Winter 'to Spring, 1973-74,. During the Winter period, 
these USMES students appeared to be spending significant amounts of time 
taking part in class discussiiajhs or pr^j^sentations (175K from line 22) and^ 
listening to/looking at their peers (13* from line 23). In the Spring, 
there seemed to be greater diversification of USMES student activities 
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across several categories,- but overalJfi^ attention to the teacher predomi- 
nated. * . ' * . 

Even though the 1974-75 sample USMES teachers dominated more^ 
class' time than expected (see Table 4, line 25), for the balance of the ' 
time their USMES students engaged in more active, creative, and aelf-^ 
directed behaviors than their controls. During the first and second 
1974-75 observation' periods, USMES students more often pursued composition 

, writing or illustrating (line 8). Throughout tine , three periods, USMfS 
students engaged more frequently in talking to r^ne another individually * 
abOMjt task-related matters (line 13^), in taking part in* small group' dis- 
*cussions about task-r el atet;! matters (line 15) gnd in giving original in- 
formation to the teacher (line 18).. - - t 

- ^1 . . The most' striking contrast betwefen USwES^tl^corrtr^ 5jtudents in- 
volved the latter group's sustained emphasis on "structured, pfesci^iptive 
activities, in their mathematics angi science classes for both years of data 
collection with STUDENTS; During '1973-74, control classes spent signifi- 
cantly larger, amounts of time in calculating (Table 3, line '6) and in pre- 
st^uctured writing (line 9)', probably for textboif^ exercises, teacher-made 
worksheets, or in workbooks. - These two activities consumed 17515 of theiP' 
observed class time during the first 1973-74 observation period, 17% dur- 
ing the second, and 19% during \the third/ The corresponding percentages . 
for the' 1973-74 sample USMES classes 'wgrer 9515, 2515 and 1515. Ffom Table 4, 
comparable figures for 197A-75 cor>trol 'felasses (summing liniss 6 and 9) 

■ were 295K, 2U% and 2AS5; for USICS classes they were 5%i' 3%, andA6*. 
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* More extensive discussion of the differences in student behavior 
Patterns betv^n USMES and control groups has been offered in the USMES 
evaluation reports^. Most 'salient to the present paper is the fact that 
the f STUDENTS observation system was capable of illuminating important dif- 
ferences in the nature of the treatments which were actually applied . 
.the experimental and control groups in field settings. Teacher qijestion- 
rraires, program monitoring forms, and cla§s information forms used, in the 
USMES. evaluation had revealed great variety in the nature -and intensity of 
the uncontrolled, self-selected t^^atments which the groups received. Asf 
signments could not be controlled by the evaluato'r. Only from the .. ^ 
STUDENTS observation data could one abstract meaningful differences in ithe ^ 

- kinds of activities'^tually pursued by. USMES versus control students.^ 
Jncreasingly, educational practitioners have voiced a neied for 
evaluation, procedures which can be applied easily and inexpensively in na- 

/ttiral classroom settings with minimal disruption to the instructional pro- 
cess. The observation procedure called- STUDENTS has teen offered as ah 
instilment of data collection which meets these practical considerations, 
and yet is sensitive enough to compare a student-centered, process- 
oriented curriculum withiiits more traditional alternatives, encouraging * 
more sensitive tesearch into relationships between student lear|ilp5 acti- 
vities and student performance,. * . ■ v. 
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Recording Form for STUDENTS to TaUy Seven 5-jnlnute Rounds of Observation 
Teacher ^ > Date Observer , 



School 



# Students 



irSMES/Control 



Unit/dubject 







1 


2 


3 


4 


5 


6 


7 




) 

Measures — 


















Counts 


• 














1 


ConscructSv 


















Assembles^ x 
















CO 


Tests/ExperimentV f 












- • 




ITIE 


Calculates I 






- 








) 


H 

O 


Records data • 
















< 


Writes composition/ill^trates 














* 


Writes (pre-sj^riictured) 












-i- 






Reads How-to Cards; Plays Tapes 


















Reads - task 


















Free 


reading, writing, drawing 












- 






o ■ . ■ ^ 
Hesses around with materials 




















Talks to another - task 


















HILDREN 


Talks to another - sqcial 
















TIONS 


Takes part in 3mall group discussion - 
task 


• 














o 


takes part in small. group discussion - 
^ social ' ^ 
















RAC 




Givbs pre-structured inf oxto teacher^ 
















ti lb 


HER 


Gives original info to teachers • task 
















M 


< 


Seeks info^rmat ion from teacher 




















.Talks to- teacher - social- 




















^i4(es part in plass discussion or pre- 
.se?Pta*tion • • 








— 










Listen/look at child 
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Llstenyi^ok at smaH groups 






• 




/ • 








Listen/look at class 
















LISTEN/ I 


Llsten/lo(^ at teacher/lecture/f 11m 
















Collecting materials/maintenance . 














• 




/ ■ ■•• ' " " . ■ 
R«8tinc/waitlnR/foolln« around \ , 

















f 

Table 2 



Descir^-lbns and Exatnples of STUDENTS Behavior Categories 



Behavior Category- Description and Exanq>le3 

illes ~ ~ 



Actlvlt 




MEASURE S > An instrjument 1$ used to measure distance 9 weighty volumi 
time^ A measurement is read from a continuous scale* 

Examples ; - v ^ 

^ . Timing with a stopwatth. * 

Measuring a bprad with a yard stick.* 
. Measuring the length of a sidewalk with^ string. 
Weighing a person on a scale* > * 

< , ' Measuring ounces of a soft drink in a measuring cup* 

Measuring amounts with measuring spoons^* 
Measuring length with a trundle wheieU 
Using a tape measure 'to measure a person's height* 
Using a' classroom clock for cif^g* ^...-....^ 
Using an eggtim^r* » Jr , ^ ' 

Measruing weather conditions with a barometer^ thermometer or' 

rain gauge* j . 

* • . ■* * ^' ' ' . 

COUNTS ! Quantities^or^frequencies are co\jnted* \ 

Examples t . f 

-Count iitg the number of preces of metal wl}lch can be picked up by 
''a magnet* ^ 

Counting the number of people going through the^ lunch line* 
' Counting the number of white Cuisenaire Rods which equal an 
orange rod* \ ^ : 

Counting the number of cars driving through an intersection* " . ^' 
Counting the nui9t!.er of squares on a piece of graph paper* 
Counting the number of times, a pencil can be sharpened* 
Counting the number of persohs with a particular eye color* 
Counting the number of ^children with freckles* 
Counting the freckles!. - 
Counting beans 9 scissors or books* 
' Counting 2* S9 5*Si or lO's* ^ 

CONSTRUCTS :: Physical components are put together to cteate a whole* Something 
is bui4t or made from scratch* 

Examples ; « 
A^chair is built* - 
/ A soft drink, is mixed* / ■ . \. 

A mustery box is made* ^ < „ . 

A mobile is made* ^ • 

An irrigation system is made* ' . ' . 

. Ingredients are mixed* , . 



ERJc ; . 



CONSTRUCTS: (cont.) 



r ' / 
✓ . 

ASSEMBLES: 



Sandals are mixed* 

Something is hammered together* 

Ah aprjm is sewed together*' 

Life-size pupj^ts are made* 

Wood tisr cut* 

Cen^ent is*mixed* 



EKLC 



Pre-cut or manufactured materiajAs are assembled* A i)lanf set o£* 
"~ ' instructions or recipe is followed* * . ' 

Examples : ■ . ' ■ -^w-. : • . 

• A plastic model iairplane, is assembled* 

> A geometric form is made flrom pre<-cut papet shapes* \ 
.A light switch is; made from electrical components* . > 
A barometer is made from a science kit* 

> Stamps are glued into a ;stamp album* ' . 
A jigsaw puzzle is put together* ; ^ 

Bones are put together* ^ ' s 

A circuit is assembled according to a plan* 

TE SIS/EXPERIMENTS ; An experiment is performed and data ^Is collected* 

■ '•■ . ' • ■ ■ • ■, '■' ^ ' '^.l • ■■ . ' ' ' 

Examples : , ■.■ 

Water is tested with litmus paper* ^ i 
- A soft^drtnk, is tested for t^ste appeal* ^ 

A circuit Is tested to see if a light will'turn on* 
A chair is tested to sfee if it is^the right size* 
^ Paper towels ar^ pulled to test strength^ , 

A blindfold taste test is run* 
' yltacks-are Scratched to determine hardness* - 
/' A qulture mold is grown on wet bread* 

A bottle a£ pop is shaken to see if it fizzes* 
' Items are dropped ^in water to see if they float*' 

CALCULATE S : Ar i thme tic is done (additioni subtractioiii multiplication and 
division)* Include math done in math workbooks* 

Examples : ;^ 
' Sums 'are. added* • 
Division is done on a Want calculator* 

Frequencies are tptaled*. - ^ ' 
; Yards are conv^erted fp fieet* ■ 
*^^ltiplication problems ar 

RECORDS DATA : A record is made of raw data* 

^ , ■ ' . ■ • " ' • . ■ ■ ' ' ' ' . . . ■ ■ 

Examples : . ' . " 

^ • The' number of people crossing an intersection is recorded* 

• * A tape recording is made of noise in a lunchroom* 
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■■■ \ ■ ■■ ' 

RECOF(gg. DATA : (cont.) 

> Pictures are taken of all the different animals for a report on 

f V the zoo. , " ^ 

A record is made of th^ number of times a die turns up three* 
^ The height' of a person ^recorded* 
^ . A map is drawn pf .an intersection*' 

. ' A record, is k^{>t^of weather information* 
^ ^ Suggestions are written on the blackboards * 
/An inventory is made* ^ 

A descrilH:ion of an jcxperimerlt is recorded* 
Physical characteristics are tallied* ^ ^ 

WRITES COMPOSITICTJ/ILLUSTRATES ;' An t)rigiT>al composition or illustration i[ 

- ' created in connection with school york o4 a 
class a^ssignment* (Inclines graphi|kg when 
^ the graph summarizes and illustrates findings*) 

' / 7 / 

^Examples : . / ^ \ 

A story^is written* • 
. k pic(:pre is painted* ' . 

k book report is written* ^ . . 

A':graph is drawn*' , * 

A ptay is written* ^ ^ • ^ ' . ^ 

.Future field trips are mapped* 

Self-portraits are draVn* " . « . 

Letters are written in connection with the class project* 
Social studies reports are written* 
^ Advertisements are written* 

Essay.tests are taken* ^ . 

Captain Cook's voyage is plotte<^on a map* 
A histogram is drayn* 

WRITES (PRE- STRUCTURED) : Writting is done in workbooks or on, worksheets* , 

Pre-structured questions are answered in writing* 

.- Examples : 

Blanks in a reading workbook are filled** ^ 

A worksheet is completed* ' # 

A poem is copied* 

Spelling words are written from dictation* 
A questionnaire is filled out* ^ 
Refereaces are copied* " . • . 

Word definitions are copied* \ 
.A true-false test is taken* 
A map is traced* 

READS HOW-TO CARDS AND PLAYS HOW-TO TAPES ; USMES How-To cards and/ or tape?^ * 

are used* 
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BEADS * TASK ; Reading is done in connection with school w.ork or a class 
assignment. * 

E xamples : ^ 
A preference book is consulted. ^ 
A text book is tead, / . , . 

A table oi^ weights'atid measures is ]^ead. ^ ^ . 
^ A magazine is read, * ^ ( 

Instructions are read. ^\ ^ 

^ A newspaper is read for wea'ther forecast and "current ^events. " 



FREE READING» WRITIn6>V'DRAWING ! Frefe time is used ^or reading^ writing 
"I \ ^ .dewing (NON^TASKS). / v 

Examples ; . ' : ^ 
A poem is written. 



or 



A vl'etter is written.- 
A^crossword puzzle is done* 
A landscape is painted. 
A novel is read. 



MESSES AROUND WITH MATERIALS : Although the child manipulates USMES <0r non- 
• . . • . ■ , ■ ■■ . USMES) materials, the purpose of his behavior. 

is not appatc^t« *^ 



E xamples ; 

Blocks are. pile^* 
Clay is pounded. 
Buzzers axe rung. 
Etc.. 



INTERACTIONS 




The child's predominant activity at the t^e of observation may be verbal 
interaction with another petson or a group of people* The observer needs to 
discriminate between the following categories. 

CHIL D vTO CHILDREN^^CATEGORIES t - 

TALK TO ANOTHER TASK : The> child talks with another child about the .task* 

TMJC TO ANOTHER ■ SOCIAL ! The child talks wi'th another child socially*^' 

TAKES PART IN SMALL^ GROUP "PTSCUSSION - TASK : The child talks in a group 
about the task* . ^ ^ 



TAKES PART IN SMALL GROUP blSClTSSION > SOCIAL : The child talks in a group 
about social f non-task topics. 



CHILD TO^ TEACHER CATEGORIES ; ; : ' • 

fePTES PRE-STRUGTURED INFORMATION TO TEACHER : i:he child responds to the . . 
'■ teacher according to a pre-structured format. ■ .: 

• gives! bRlGINAL INFORMATION TO TEACHER ; ° The child talks with or to the 

teacher ab'out the task. ' ' 

Seeks INFtiMlATiON from teacher ; The child seeks information jf^om the 
teacheii questions the teacher. : : - ' • * * '; 

TALKS TO -TEACHER - SOCIAL ; The child talks with the teacher socially. 

^ ■ ■ ■ ' ' . 

• TAKES PART IN CLASS DISCUSSION OR gRESENTATION !?/ The child takes part in 
' a class discussiori or gives a presentation to the class. 



LiSTEN/LOOK ; ' 

> LISTEN/LOOK > AT CHI lit) ! The chlid attends to another child. . . • . 

: T. TSTEN/LOOk AT SMALL GROUP ! The 'ihild pb^^ a group setting. 

L ISTEN/Lo6k> ATXLASS t The child observes, looks oh durinjg a total* class - 
.*^activtty.i - v .• ' , ■ ■. ■ • 

;LISTEN/L00k > AT TEACHER/ LECtURE/FILH t The child attends to a. teacl^er, a . 
■lecture or. a film.^\"^ • -''^'.'[.^ 'y' 



' COLlECmNd MATsteLSy Materiatis- equipirtent is 

•maitttained.- /. -rr^- . V'^. ■ 



Examijles: . . 



/I pencil is ^sharpehedk; : /'V • ■ V':"' 

Supplies for paintinj^ a^reV^^^^^^^ , 



RESTING/WAITING/FOOLING AROUND/ ATTENDING TO SOMETHING OU TSIDE THE CLASS: 
'" The child is not actively involved learning q s • 

The chtlcl is phased otut or distracted. . . 



^ ■ • . •• Table. 3 ■ 

STDDENTS Observation Data. from the 1973-74 USMES Evaluation 



Observation Pierlod. 


FALl 


WINTER 


SPRING 




USMES 


USMES USMES 


USMES 


Treatment Group ' 


Imp. Control 


Dev. Imp. Control 


Imp. Control 


Number of Classes 




a4^ ay (s) 


(5) (3) 



)itftiegory of Student Behavior 



Percentages of Tallies in Each Category 



1. Measures .. 

.■ ■ . ' - . . ^ , 

2. . Counts 

3.. Constructs ' ' 

4. Assembles ■ 

5. Tests/Experiments 

6. Calculates 

7. Records Data 

8. Writes/Illustrates 

9. Writes (pre-structured) 

10. R^ads How-To-Cards 

11. ReadsrTask 

12. Free lleadlng, Writing, Drawing 

13. 'Messes ground with Materials 

14. Talks to Another-Task 

15. Talks to Another-Soclal 

16. Small Group-Task ' 

17. Small Group-Social 

18. Gives Pre-structured Info to 

Teacher ' 

19. Gives Original Info to Teacher 

20. ^ Seeks Info from Teacher ^ 
J 2U Tal^s to Teacher, Social 

22. Takes Part In Class Discussion, 

Presentation. 

23. . Listen/Look at Child 

24. LlstiBn/Look at Small Group ^ 

25. Listen/Look* at Ct«ss 
26» Listen/Look^ at Teacher' ^ 
27. Collecting^ate^al/Maintenance 
28* R<>iii^lVg/WAltlng/FoollnjB Around 









0 4 


7-9 


0-4 


4.5 


0-0 


18.4 


0.8 


8.2 


11.7 


6. 2 


1.8 


0.2 


0.2" 


0.4 


5.7 


0.2 


0.0 


0. 1 


1.9 


6.5 


0.9 


1.1 


1.2 


2.2 


1.0 


3.8 


4.7 


2.3 


0.3 




1.9 


0.6 ; 


1.5 


3.3 


6.8 


2.9 


3.5 


0.2 


0.4 


4.9 


11.2 


7.3 


1.9 


1.4 


7.2 




2.6 


14.1;^ 


31.0 


- 2.6 


4.6 


2,2 


2.6 



1.8 
1.2 
11.6 
3.0 
1.3 
0.9 
1.1 
5.6 
2.1 
0.f» 
2.8 
3.4 
1.3 
2.8 
1.5 
12.4 
0.1 
0.5 

1.3 
2.0 
0.3 
6.7 

4.8 
1.2 
7.2 
13.2 
4,2 
4.9' 



2.0 




6.9 


0 0 




1-0 


0,0 


0-0 


2^ 1 


5.7 


1.9 


10.6 


6." 5 


0.2 


2. 5 


4.1 


0.0 


6.1 


0.0 


0.0 


4.7 


0.8 


2.2 


i.2 


0. 1 


• 0.9 


3.2 


3.5 


3" 2 


7.2 


0.0 


1.4 


0.6 


0.0 


1-0 


2^3 


2.2 


1.7 


1.7 


2. 7 


2.3 


0.0 



17:1 



13, 
2V 

'21. 
. 0. 
6. 



4.4 

4.2 
1:2 
0.4 
30.1 
2.0 
9.1 



2.5 

0.9 

2.8 

1.8' 

4.0 

0.6 

2.3 

3.9 

0.0 

0. 1 
4.3 
1.2 

1. A 
4.6 
4.1 

1.*. 
0.0 
1.8 

6.1 
2.6 
0.1 
8.7 

. 2.7 
0.3 
0.5 



6.0 
0.0 
0.6 
0.0 
0.0 

T2.7 

0.0 
0.0 
6.3 
0.0 
0.0 
0.6 
O.p 
3.9 
8.0 
0.0 
0.0 
9.4 

0.0 
1.2' 
0.0 
10.7 

2.3 

•«.0 
1.9 



28.4 .26.3 

4% 2.8 3.5 

3:^.: 1 1.1 



Total ,Pereen«iges 



lOO.l . 



100.2 



100.0 100.0 99.8 



.100.2 99.9 



STUDENTS Observation Data from the .19 7A-75 USllES 'Evaluation 



Observationr Period . ^ 


Beginning 


Middle 


'End 


of Unit 


. of Unit 


of Unit 


Troatment Group 


U?WES jconwrol 


USME3 


Control 


USMES 1 Control 


Number, of Classes . ^ 


(:-3) (27) 




(24) 


(.22^ -(24) 



Category' of Student Behdvlor 



Percentages of T all ies in Each C ategory 



!• Measures 
2. . Counts. ".' - - 

3« /'Construats/ ssembles 

4. Graphs . 

5. Tests/ xperiments 
. 6, Calculates 

7« Records Data 

8« Writes composition/ illustrates . 

9. Writes (prestructured) 

10. Reads How-to-cardsf^^lys tapes 
IK . Reads (prestructured) 

12. Tree* Reading 

13* Talks to another- task 

14« Talks to ar.other*social 

15« Takes part in small group da scusslon- task 

16; Takes part in small ^roUp discussion-social 

17« Gives prestructured inf ormartion to teacher 

Gives original information to teacher 

i9» Seeks iql^ormat ion from teacher 

20* talks to teacher<^social 

,21* Taljces part in class discussion or pre* 
sentation 

22. Lijsten/look at child , 

23* Lis ten/ look at small group 

24, tisten/look at class 

25* Listen/look at teacher 

26* Listen/.look at film or AV materials 

27* Collecting tnaterials/maintenanpe. 

28. -Resting/waiting ^ 

29» Fooling around^. 



0. 

(I. 
( . 
^, 
J « 
1. 
(• 

2. 

2 m 

C. 

5.. 

4. 
2. 
1. 
19. 
0. 
2. 
6. 
3. 



• 6 
.1 

• 0 
.2 

• 3 

• 9. 

• 4 

• 2 

• 7 
.3 
.7 

6 
4 
6 
8 

? 
I 
8 

4' 

3. 
5 
8 
4 

6 
4 
0 



O.l 
. Ul 
C.6 
I.O 
0.6 
17,2 
UO 
2.9 
i ! ,7 
0.0 
6.0 
l.l 
3,4 
5.2 
1.5 
0.6 
2.8 

y 

3.7 
0.1 
1.4 

U.6 
0.9 « 
0.3 
16. 5 

2.2 
7.0 
2.9 



J 6.1 



1.6 

0.5. 

9.4 

1. 1 

2.7 

0.4 

:.? 

7,4 
*2.5 
0.0 
1.1 
1.5 
5.7 
3.1 



0.8 

0. B 
3.0 

1. '7 
0.3 
5.9 

5.5 
1.8 
4. 1 
1 7.7 
0.0 
3.6 
5. 5 
3. 3 



0.0 
0.1 
0.0 
0.0 
■ 1.5 
13.6 

2.2 
10.6 
0.0 
13.4 
. l.O 
3.7 
' 3.4 
1.3 
1.4 
1-5 
0.8 
2.1 
0.1 
1.8 

6.2 
0.7 
1.3 
20.4 
3>5 
1.9 
4.6 
2.7 



7.5 
1.-7 

1.9 
20.1 



5 



0.2 
0.4 
0.0 
0.0 
2.5 
15.1. 
1,7 
0.9 
9.2 
0.0 • 
5.8 
O.l 
3.7 
3.6 
1.8 
1.6 
1.8 

^•1 
2.4 

0.0 
2.4 

9.5 
1.0 
3.2. 
19.4, 



0.2 ^^/^2jKr 

3.9 '1.7 

7.9 5.4 

4.6 3.2 



TO 



PERCENTAGES 



100. 



100. 0 



99.8 



99.9 



100.2 



99.8 



